1. Field of the Invention
The present invention relates to an image processing apparatus for obtaining a high-quality image by processing image data captured by a CCD (charge-coupled device) area sensor using a complementary-color filter. The image processing apparatus may be used in a digital camera (e.g., an electronic still camera) or the like.
2. Description of the Related Art
Conventionally, it has been known that a complementary-color filter has a higher transmittance, and obtains a higher number of G-components obtained in extracting a luminance component, than those of a primary-color filter. Therefore, when a complementary-color filter is used in a CCD area sensor, a high-sensitivivity image data can be obtained.
An exemplary CCD area sensor with a complementary-color filter will be described below by taking a color video camera as an example. FIG. 15 shows a complementary-color filter having pixel units. The complementary-color filter is provided at a light-receiving element surface side of the CCD area sensor. The CCD area sensor is a color difference progressive type CCD, in which image data is read out in such a manner that a color signal is obtained as a color difference signal every scanning line and alternately between an odd-numbered field and an even-numbered field. Specifically, the CCD area sensor is scanned as follows. Referring to FIG. 15, first, an odd-numbered field is read out in such a manner that the value of an odd-numbered pixel is added by the value of a subsequent even-numbered pixel (numbering is conducted in a vertical direction from top to bottom). Thereafter, a subsequent even-numbered pixel is read out in such a manner that the value of an even-numbered pixel is added by the value of a subsequent odd-numbered pixel, unlike the odd-numbered field. For example, in the case of an odd-numbered field, a signal is obtained on a scanning line {circle around (1)} in the order of Ye+Mg and Cy+Gr, and a signal is obtained on a scanning line {circle around (2)} in the order of Ye+Gr and Cy+Mg. In the case of an even-numbered field, a signal is obtained on a scanning line {circle around (1)}′ in the order of Mg+Ye and Gr+Cy, and a signal is obtained on a scanning line {circle around (2)}′ in the order of Gr+Ye and Mg+Cy. Hereinafter, such a signal is also referred to as a scanning line input when it is input to a subsequent stage of an image processing apparatus.
A relationship between Ye (yellow), Cy (cyan), Mg (magenta), and Gr (green) of the complementary-color filter of FIG. 15, and three primary colors, i.e., R (red), G (green), and B (blue) is ideally represented by:Ye=R+G, Cy=G+B, Mg=R+B, Gr=G  (6).
The thus-read image data is processed by a CDS (Correlated Double Sampling) circuit for reducing noise, and then by an AGC (Automatic Gain Control) circuit for adjusting gain. The resultant image data is converted to digital image data by an A/D conversion circuit having a resolution of 10 bits, for example. The resultant digital image data is input to an image processing apparatus in which the digital image data is subjected to various kinds of image processing. Finally, the resultant image data is output as a video signal to be displayed. The CDS circuit, the AGC circuit, and the A/D conversion circuit are included in a scanning line input section 410 of a color difference progressive type CCD in FIG. 16. This conventional image processing apparatus will be described below in more detail with reference to “Digital Signal Processing System of Single-Chip CCD Camera for Business Use”, Toshiba Review, 1994, Vol. 49, No. 1.
FIG. 16 is a block diagram showing an exemplary configuration of an image processing apparatus (signal processing system) in a conventional color video camera. Referring to FIG. 16, a signal processing system 400 of a color video camera includes: a horizontal-vertical contour emphasizing section 401 which receives a scanning line input of a color difference progressive type CCD; a low-pass filter (LPF) 402 which receives the scanning line input; a gamma correction section 403 which receives an output of the low-pass filter (LPF) 402; a color separation section 404 which receives the scanning line input; a white balance adjusting section (WB) 405 which receives an output of the color separation section 404, a gamma correction section 406 which receives an output of the white balance adjusting section 405; a color difference matrix section 407 which receives an output of the gamma correction section 406 and outputs color difference signals Cr and Cb; a constant luminance processing section 408 which receives outputs of the gamma correction section 403 and the color difference matrix section 407; and an adder 409 which receives outputs of the horizontal-vertical contour emphasizing section 401 and the constant luminance processing section 408 and outputs a luminance signal Y.
The horizontal-vertical contour emphasizing section 401 receives an output from the scanning line input 410 of the color difference progressive type CCD and performs horizontal-vertical contour emphasizing processing. The horizontal contour emphasizing processing is performed for neighboring image data on the same scanning line. For example, (Ye+Mg)−(Cy+Gr) is calculated on the scanning line {circle around (1)}. Vertical contour emphasizing processing is performed for image data on neighboring scanning lines in a field. For example, (Ye+Mg)−(Ye+Gr) is calculated on the scanning lines {circle around (1)} and {circle around (2)}.
The low-pass filter (LPF) 402 receives an output from the scanning line input section 410 and cuts out a high-range luminance component to output a broad frequency luminance component Y1. The broad frequency component Y1 is calculated in groups of four pixels for each scanning line in accordance with expression (7) below. It should be noted that the center of the broad frequency component Y1 is the center of four pixels in each scanning line. Calculation of expression (7) is carried out in groups of four pixels on the scanning line {circle around (1)} of FIG. 15.Y1=Ye+Cy+Mg+Gr  (7)
The gamma correction section 403 receives the broad frequency luminance component Y1 output from the low-pass filter 402 and performs gamma correction. By the gamma correction, an image is modified so as to fit characteristics of a display or printer from which the image is output.
The color separation section 404 performs color separation as follows. Ye-, Cy-, Mg-, and Gr-components on the scanning line {circle around (1)} are assumed to be the same as respective Ye, Cy, Gr and Mg-components on the scanning line {circle around (2)} neighboring the scanning line {circle around (1)}, although each pair of components have different positions. Under this assumption, Ye-, Cy-, Mg-, and Gr-components are separated. The separate Ye-, Cy-, Mg-, and Gr-components are converted to three primary colors, i.e., R-, G-, and B-components in accordance with expression (6) above in groups of eight pixels, for example.
Based on the R-, G-, and B-components obtained by the conversion of the color separation section 404, the white balance adjusting section (WB) 405 adjusts white-balance in accordance with the color temperature of illumination so as to correct the color of an image.
The gamma correction section 406 subjects image data output from the white balance adjusting section (WB) 405 to gamma correction.
The color difference matrix section 407 calculates a low-frequency luminance signal Y2 (also called a constant luminance signal Y2) based on the R-, G-, and B-components in accordance with expression (8) below, and calculates color difference signals Cr and Cb in accordance with expression (9) below.Y2=0.3R+0.59G+0.11B  (8)Cr=R−Y2Cb=B−Y2  (9)
The constant luminance processing section 408 replaces a low frequency portion of the broad frequency luminance component Y1 which has been subjected to gamma correction in the gamma correction section 403 with a low-frequency luminance signal Y2. This processing is referred to as constant luminance processing.
The adder 409 adds a horizontal-vertical contour emphasizing processing signal output from the horizontal-vertical contour emphasizing section 401 to a luminance signal output from the constant luminance processing section 408, and outputs the resultant signal as a luminance signal Y.
As described above, the signal processing system 400 of the color video camera reads out image data by adding pixel data on two scanning lines as shown in FIG. 15, so that a resolution is reduced. Nevertheless, a problem substantially does not arise, since the resolution of a display is as low as the resolution of the video camera. In the case of a digital still camera requiring a higher resolution, the signal processing system needs to read out pixel data for every scanning line.
For a complementary-color filter array as shown in FIG. 15 or 2, an actual sampling frequency is set to fs=1/Δx=1/Δy where ΔX represents a width of a pixel (pixel pitch) in a horizontal direction (x-direction), and Δy represents a width of a pixel (pixel pitch) in a vertical direction (y-direction).
According to sampling theorem, the highest frequency of spatial frequencies contained in an original image, which can be restored, is half the sampling frequency fs (=1/Δx=1/Δy). Therefore, frequency components higher than the highest restorable frequency fs/2 appear as noise.
To avoid such a problem, an optical low-pass filter (anti-aliasing filter) is attached to a CCD area sensor. The optical low-pass filter cuts out frequency components higher than or equal to fs/2. Unfortunately, the optical low-pass filter is not ideal, so that frequency components lower than or equal to fs/2 are attenuated. Referring to FIG. 8, graph a shows a frequency characteristic of an ideal low-pass filter (for cutting out frequency components higher than or equal to fs/2). However, an actual low-pass filter has a frequency characteristic as shown by graph b. Graph c shows a frequency characteristic of a desired compensation filter for restoring high-range luminance components which are attenuated by the low-pass filter to approach the ideal frequency characteristic represented by graph a. In the present invention, a compensation filter having a frequency characteristic substantially represented by graph D1 is used to newly extract a middle-high-range luminance component and combine it with middle and high-range luminance components at a predetermined ratio. The term “middle-high-range” as used herein refers to an intermediate range between a middle range and a high range in a region less than or equal to the sampling frequency fs.
When for each pixel of the complementary-color filter array of FIG. 2, three color components are estimated using interpolation, high-range luminance components are attenuated. Therefore, compensation of the attenuated high-range luminance components is essential so as to produce a sharp image. Typically, as shown in FIG. 9, such compensation is carried out by a combination of a frequency characteristic of a middle-range luminance component compensating filter (graph d) and a frequency characteristic of a high-range luminance component compensating filter (graph c). In FIG. 9, graph b represents a total frequency characteristic after a low-pass filter (anti-aliasing filter) and interpolation, and graph a represents an ideal frequency characteristic of an entire image processing system including compensation.
In the above-described conventional technique, a middle-range luminance component and a high-range luminance component are compensated for by the frequency characteristic curve (graph d) of the middle-range luminance component compensating filter and the frequency characteristic curve (graph c) of the high-range luminance component compensating filter of FIG. 9. A maximum amplitude of graph d is positioned at an angular frequency ω=π/2 (corresponding to fs/4). A maximum amplitude of graph a is positioned at an angular frequency ω=π (corresponding to fs/2). Therefore, the middle-range luminance component is compensated for by graph d having the maximum amplitude at an angular frequency ω=π/2 while the high-range luminance component is compensated by graph c having the maximum amplitude at an angular frequency ω=π.
However, when high-range luminance components in an image are emphasized by the high-range luminance component compensation in contour emphasizing processing for sharpening an image, noise components are also emphasized. Therefore, all components having an angular frequency ω higher than π (corresponding to fs/2) are noise. As a result, when the resolution of an image is increased, noise and jaggy components are made conspicuous. Jaggy refers to one kind of noise which is substantially the most conspicuous of various kinds of noise. Jaggy (or zip noise) in the shape of steps appears at a contour portion.